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For many educators it has been challenging to meet the Accreditation Council for Graduate Medical Education's 
requirements for teaching systems-based practice (SBP). An additional layer of complexity for educators is 
evaluating competency in SBP, despite milestones and entrustable professional activities (EPAs). In order to 
address this challenge, the authors present the results of a literature review for how SBP is currently being 
taught and a series of recommendations on how to achieve competency in SBP for graduate medical trainees 
with the use of milestones. The literature review included 29 articles and demonstrated that only 28% of the 
articles taught more than one of the six core principles of SBP in a meaningful way. Only 7% of the articles 
received the highest grade of A. The authors summarize four guiding principles for creating a competency- 
based curriculum that is in alignment with the Next Accreditation System (NAS): 1) the curriculum needs to 
include all of the core principles in that competency, 2) the objectives of the curriculum should be driven 
by clinical outcomes, 3) the teaching modalities need to be interactive and clinically relevant, and 4) the 
evaluation process should be able to measure competency and be directly reflective of pertinent milestones 
and/or EPAs. This literature review and the provided guiding principles can guide other residency educators 
in their development of competency-based curricula that meets the standards of the NAS. 
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For many educators, it has been challenging to meet 
the Accreditation Council for Graduate Medical 
Education's (ACGME) requirements for teaching 
the competency on systems-based practice (SBP) (1). 
Unveiled in 1999, the goal of the ACGME's Outcome 
Project was to move from basic knowledge and process 
measures to an accreditation system that focused on 
competencies. The multi-phase project began by inviting 
programs to define specific behaviors that would reflect 
the competency, as they pertained to that specific spe- 
cialty. Phase II (2002-2006) of the project concentrated 
on refining the definitions of the competencies and their 
respective assessment tools. The goal of Phase III (2006- 
2011) was full integration of the competencies and their 
assessments into learning and clinical care. The Out- 
comes Project's goal was clear: 'Residency programs are 
expected to phase in assessment tools that provide useful 
and increasingly vahd, reliable, evidence that residents 
achieve competency-based educational objectives' (2). Most 



recently, to assist educators in reaching the Outcomes 
Project's goal, the concepts of developmental milestones 
(3) and entrustable professional activities (EPAs) (4) have 
been established. 

The educational outcomes are grounded in the six core 
competencies: 1) patient care; 2) medical knowledge; 3) 
practice-based learning and improvement (PBLI); 4) 
SBP; 5) professionalism; and 6) interpersonal skills and 
communication. Of these competencies, PBLI and SBP 
are the newest additions. SBP has six core principles and 
is defined as an 'awareness of and responsiveness to 
larger context and system of health care and the ability to 
effectively call on system resources to provide care that is 
of optimal value' (1). Specifically, all residents need to 
demonstrate the ability to: 

1) work effectively in various health care delivery 
settings and systems relevant to their clinical specialty; 
2) coordinate care within the health care system relevant 
to their clinical specialty; 3) incorporate considerations of 
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cost awareness and risk benefit analysis in patient and/or 
population-based care, as appropriate; 4) advocate for qual- 
ity patient care and optimal patient care systems; 5) work 
in interprofessional teams to enhance patient safety and 
improve patient care quality; and 6) participate in identify- 
ing system errors and implementing potential solutions. 

The goals of the ACGME, as evidenced by Phase II of 
the Outcomes Project, are for residency programs to use 
dependable measures to assess residents' competencies 
and, subsequently, to use those individual assessments to 
evaluate the educational effectiveness of the residency as 
a whole. As is clearly defined, these abilities are expected 
to be specific to each specialty (2) - and each competency 
should be sufficiently unique to merit its own category. 
Yet, data have shown that residencies are having diffi- 
culties measuring the individual competencies as inde- 
pendent constructs, and assessment tools often measure 
'behaviors that consistently map onto three or more of 
the general competencies' (5). As a result, educators con- 
tinue to struggle with the creation, definition, and cate- 
gorization of six separate competencies. Furthermore, 
uncertainty continues to persist as to which assessment 
tools are most valid. Because true competency extends 
far beyond simple knowledge, educators are looking for 
measures that can assess 'authentic' behaviors of real-life 
clinical context. To date, only a 'few models offer clear 
and direct routes' for competency assessments without 
having measurement pitfalls (1). 

Despite often being educators' lowest priorities, this 
challenge especially exists for SBP and PBLI. Indeed, 
a survey of family medicine program directors rated SBP 
as the lowest importance of all the competencies, with 
70% reporting no method to evaluate SBP (6). A major 
challenge is that SBP is truly a skill-based activity that 
cannot be validly captured with simple knowledge-based 
assessments. In response to these and other similar edu- 
cational challenges, the ACGME developed 142 discrete 
curricular milestones that map to the six specific core 
competencies (3). 

Even with the development of various educational 
constructs (e.g., competencies, milestones, EPAs), resi- 
dency programs continue to struggle with the use of 
milestones and/or EPAs in their curricular goals, objec- 
tives, and observational evaluative tools (4) - and little 
related information has been reported specific to SBP. 
Hence, we explored how SBP is being integrated into 
residency education and whether or SBP measures were 
in alignment with the Next Accreditation System (NAS). 
We began by reviewing our own internal medicine resi- 
dency program, and found that our SBP curriculum 
and its assessments measured associated knowledge and 
attitudes - but probably not true competency (7). It was 
obvious, then, that we needed to evaluate in far more 
greater detail the six components of SBP as they apply 
to internal medicine residency programs. To move from 



process measures to hard outcomes like SBP-related 
developmental milestones or EPAs, we conducted an 
extensive review of the literature on SBP in internal 
medicine residency programs. The results of that litera- 
ture review are presented here, along with recommenda- 
tions on how to achieve competency and best use NAS 
evaluative measures in SBP for graduate medical trainees. 

Methods 

Search strategy 

We reviewed literature published from 1999 to April, 
2012 present using PubMed, Google Scholar, and ERIC 
(Fig. 1) - using the following key search terms: 1) core 
competencies; 2) system-based practice; and 3) graduate 
medical education. For this search 1,158 titles were 
generated. Of the original 1,158 titles reviewed, 179 titles 
remained after duplicates, books and titles that named a 
specialty other than internal medicine were eliminated. 
The 179 respective abstracts were reviewed and 37 full 
articles were chosen after applying our inclusion criteria. 
To these 37 articles, 6 more articles, gleaned from a bib- 
liographic review, were added. We also supplemented our 
pool with seven articles from the ACGME's Outcomes 
Project web-based reference guide. After carefully reviewing 
these 50 articles, we excluded 21 which, despite their titles, 
did not align with our predetermined inclusion criteria. 

Inclusion and exclusion criteria 

All abstracts and articles were included if they contained 
either a description of the curriculum and/or the asso- 
ciated evaluation methods. The described curriculum 
had to be focused on the core competency of SBP within 
US internal medicine residencies. Articles were excluded 
if they: 1) were in foreign medical programs; 2) were pri- 
marily intended for medical students/fellows; 3) focused 
on more than two core competencies; or 4) involved 
learners in other (non-internal medicine) specialties. 
Articles discussing SBP and one other core competency 
(patient care, medical knowledge, PBLI, professionalism, 
or interpersonal skills and communication) were included. 
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Fig. 1. Search strategy for articles on systems-based practice. 



2 

(page number not for citation purpose) 



Citation: Med Educ Online 2014, 19: 24441 - Iittp://dx.doi.org/10.3402/meo.v1 9.24441 



From SBP process measures to true competency 



However, opinion pieces, review articles, and consensus 
conferences were excluded - as were papers from non- 
peer-reviewed journals. 

Articles 

As stated, the initial search resulted in 1,158 references 
(Fig. 1). After the above search strategies, removal of 
duplicates, and a thorough screening using inclusion and 
exclusion criteria, 29 articles were included in the sys- 
tematic review (8-36). Given the exploratory nature of 
this study, we had little basis for organizing these studies 
(5). A subsequent review of the literature helped define 
SBP and its principles and implied abilities - from which 
six categories were developed based on Graham's taxon- 
omy of SBP (1) (Table 1). The primary aim of the re- 
viewers, then, was to categorize each article into one of 
the six groups; however, several articles contained no 
single primary theme, but instead meaningfully addressed 
more than one of the six SBP principles and were sub- 
sequently grouped under more than one category. When 
this occurred, all three authors agreed to do so. 

A quality index, based on definitions from the 
ACGME's requirements for achieving competency, was 
established to 'grade' articles. This index ranged from A 
to D (with A being the highest) and was applied to 
descriptions of the curriculum and evaluation methods. 
Because our literature review went back to 1999, the 
grading system did not include the criteria for milestones 
or EPAs - which emerged much later (3, 4). The definition 
of each grade is listed below - and centers primarily on the 
measured focus of the curriculum and evaluation: 

1. Patient outcome(s); 

2. Behavioral change by the provider (mostly closely 
hnked to milestones or EPAs); 



3. Perceptions, knowledge, or attitudes of providers 
and/or patients; and 

4. Description of content only - no evaluation was 
completed. 

Initial classification of category and quality was done 
independently by two of the authors (JM and EP). A 
third reviewer (CH) was used to reconcile disagreements. 
All reviewers are faculty members involved in residency 
education. One (JM) is a core faculty member who co- 
directs two residency educational units, whereas the other 
(EP) leads the primary care research block. The third 
reviewer (CH) is the primary care program director. 

Results 

After systematically culling the larger sample, the sys- 
tematic review included 29 articles (Fig. 1). From these, 
the two reviewers exhibited 66 (19/29) and 90% (26/29) 
agreement on their assignment of category and grade, 
respectively. 

Only one internal medicine graduate curriculum, re- 
presented in 2 of the 29 articles (7%), incorporated all 
the six principles of SBP into their described curriculum; 
eight (28%) articles included curricula covering more 
than one aspect of SBP in a meaningful way. The 
majority of articles (45%, or 13/29) were classified as 
'system evaluator' - concentrating on the principle of 
'identifying system errors and implementing potential 
solutions', or what is most readily labeled as quality 
improvement (Table 2). Few articles formally addressed 
coordination of care or patient advocacy. 

Only 7%o (2/29) were graded A - the highest quality. 
These articles represented the ACGME's 'gold standard' 
of assessing mastery of a competency via patient out- 
comes. These two articles reflect examples of what most 



Table 1. Categories from SBP definition 



ACGME definition 



Category/principle 



Examples/topics 



Worl< effectively in various healtli care delivery settings 
and systems relevant to their clinical specialty 

Coordinate care within the healthcare system relevant 

to their clinical specialty 
Incorporate considerations of cost awareness and risk 

benefit analysis in patient and/or population-based 

care, as appropriate 
Advocate for quality patient care and optimal patient 

care systems 
Work in interprofessional teams to enhance patient 

safety and improve patient care quality 
Participate in identifying system errors and 

implementing potential solutions 



System consultant Didactics on insurance, managed care, community 

resources, or experiences in different settings like nursing 
homes, hospice and/or senior centers 

Care coordinator Reviewing prior records, scheduling appointments 

Resource manager Billing, coding, cost containment, care management 



Patient advocate Policy, patient education, translators 

Team collaborator Multi-disciplinary rounds- pharmacy, nutrition etc. 

Enhanced communication skills 
System evaluator Ql projects, PDSA cycles, sign-outs 
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Table 2. SBP article categorization 



Category Total% (n =29f Article 



System consultant 


31% 


(9) 


Allen E et al. (2005), David R and Reich L (2005), Esl<ildsen M (2010), Hingle S et al. (2009), Hingle 
Setal. (2011), Nagler Aet al. (2010), Peters A etal. (2008), Tartaglia K et al. (2010), Turley C et al. 
(2007) 


Care coordinator 


7% 


(2) 


Hingle S et al. (2009), Hingle S et al. (201 1) 


Resource manager 


28% 


(8) 


Crites G and Schuster R (2004), Englander R et al. (2010), Hingle S et al. (2009), Hingle S et al. 
(201 1), Korn L et al. (2003), Kravet S et al. (2001), Nagler A et al. (2010), Perez J et al. (2009) 


Patient advocate 


10% 


(3) 


Hingle S et al. (2009), Hingle S et al. (201 1), Tartaglia K et al. (2010) 


Team collaborator 


31% 


(9) 


Daniel D et al. (2009), Elser A and Connaughton-Storey J (2008), Hingle S et al. (2009), Hingle S 
et al. (201 1), Kirsh A and Aron D (2006), Nabors C et al. (201 1), Sehgal N et al. (2008), Sherman S 
et al. (2007), Ziegelstein R and Fiebach N (2004) 


System evaluator 


45% 


(13) 


Allen E et al. (2005), Amin A and Rucker L (2004), Daniel D et al. (2009), Gakhar B and Spencer A 
(2010), Hingle S et al. (2009), Hingle S et al. (2011), Leenstra J et al. (2007), Oyler J et al. (2008), 
Peters A et al. (2008), Reznek M et al. (2010), Tomolo A et al. (2005), Wittich C et al. (2010), 
Zupancic M et al. (2010) 



'The % of articles that covered that category. 



educators set out to achieve when developing curricula - 
and go beyond the typical assessment of learners' knowl- 
edge of content. Interestingly, none of the articles, even 
those published after 2009, stated specifically how mile- 
stones and/or EPAs were addressed in their curricular 
design and/or evaluation process, if they occurred. 
However, both of the two 'A' articles were among those 
more recently written - suggesting that educators may be 
refining curricula to mirror more closely the goals of the 
Outcomes Project and NAS (Table 3). 

Study and search limitations 

Our search included only published, peer-reviewed arti- 
cles - each of which contained its own set of limitations 
and possible biases. Also, the articles did not use uniform 
definitions or terms, and exclusively focused on internal 
medicine residency programs. Lastly, because existing 



Tctble 3. Quality index grade of reviewed literature 



Grade 


Total % (n=29) 


Article 


A'^ 


7% (2) 


Daniel D et al. (2009), Englander R et al. (2010) 




24% (7) 


Gakhar B and Spencer A (2010), Hingle S et al. (2011), Leenstra J et al. (2007), Nabors C et al. (2011), Oyler J 






et al. (2008), Sherman S et al. (2007), Zupancic M et al. (2010) 




55% (16) 


Allen E et al. (2005), Crites G and Schuster R (2004), David R and Reich L (2005), Eiser A and Connaughton- 






Storey J (2008), Kirsh A and Aron D (2006), Kravet S et al. (2001), Korn L et al. (2003), Nagler A et al. (2010), 






Perez J et al. (2009), Peters A et al. (2008), Reznek M et al. (2010), Sehgal N et al. (2008), Tartaglia K et al. 






(2010), Tomolo A et al. (2005), Turley C et al. (2007), Ziegelstein R and Fiebach N (2004) 




14% (4) 


Amin A and Rucker L ( 2004), Eskildsen M (2010), Hingle S et al. (2009), Wittich C et al. (2010) 



^The described curriculum and its evaluation measure assessed an actual change in patient outcomes. 

''The described curriculum and its evaluation measure assessed an actual behavioral change amongst the provide. 

''The described curriculum and its evaluation measure assessed a provider and/or patient perception, attitude or knowledge change. 

''Description of curriculum alone. 



rating criteria had not been previously applied to medical 
education literature, our development and use of the 
present coding scheme is somewhat novel and, as such, of 
unknown vaMdity. 

Discussion 

This comprehensive literature review suggests that efforts 
are underway to incorporate SBP into internal medicine 
residency education - with isolated examples geared 
toward evaluating competency in this arena. Yet, the 
data also indicate that residency programs are lagging 
behind the Outcomes Project's goal of full integration by 
2011 - the largest barrier seemingly to be the design of 
curricula and operational measures that can indepen- 
dently assess each of the six competencies, even with the 
inclusion of milestones. This review shows that despite 
much curricular development, there appears to be little 
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agreement and data regarding best practices in asses- 
sing learner competence - especially with regards to 
SBP and its six core principles. Furthermore, areas that 
require behavior or performance-based assessment (e.g., 
milestones, EPAs) are, in contrast to knowledge and 
written assessments (37), still among the least developed 
areas. 

Before moving ahead to Phase IV (2011 and beyond) 
of the Outcomes Project, educators must verify that the 
first three phases have been successfully completed. This 
review of the literature has shown that in an attempt 
by the ACGME to make definitions widely applicable 
to all programs and institutions, there exists 'lack of 
enough specificity to move our understanding towards 
what exactly it is residents need to demonstrate in order 
to indicate proficiency in SBP' (38). Each article de- 
scribed different methods and approaches, and the data 
show that most curricula concentrate efforts on only one 
of the six principle areas of SBP. As a result, there is 
much progress to make before SBP can be comprehen- 
sively taught and assessed as a core competency. 



Lessons learned and guiding principles 

Despite the abovementioned challenges, educators should 
focus on the practical value of proposed educational 
changes. In the next steps, as an educational community, 
we must agree upon a basic model of teaching and 
assessing each competency that also provides room for 
divergence and innovation. Keeping in mind the 'gold 
standard' of improved patient care, we must move towards 
hard outcome measures of competency - for example, 
documenting milestones and completion of EPAs (39, 40). 

We offer four guiding principles that might inform 
educators when creating or restructuring SBP curricula. 
The principles were applied to our own SBP program 
efforts (5), but can just as easily be used in any curriculum 
that will address any of the six core competencies of 
patient care, medical knowledge, PBLI, SBP, profession- 
alism, and interpersonal skills and communication. 

1. The curriculum must include all core principles of 
the specified competency. 

In a hypothetical example of reformatting a SBP 
curriculum, the knowledge content should first be 



Table 4. Linking SBP category to milestones 



SBP category/principle 



Milestone 



System consultant 
Care coordinator 

Resource manager 



Patient advocate 
Team collaborator 



System evaluator 



1. Understand unique roles and services provided by local health care delivery systems. 

2. Manage and coordinate care and care transitions across multiple delivery systems, including ambulatory, 
subacute, acute, rehabilitation, and skilled nursing. 

3. Reflect awareness of common socioeconomic barriers that impact patient care. 

4. Understand how cost-benefit analysis is applied to patient care (i.e., via principles of screening tests and the 
development of clinical guidelines). 

5. Identify the role of various health care stakeholders including providers, suppliers, financiers, purchasers, and 
consumers and their varied impact on the cost of and access to health care. 

6. Understand coding and reimbursement principles. 

7. Identify costs for common diagnostic or therapeutic tests. 

8. Minimize unnecessary care including tests, procedures, therapies, and ambulatory or hospital encounters. 

9. Demonstrate the incorporation of cost-awareness principles into standard clinical judgments and decision 
making. 

10. Demonstrate the incorporation of cost-awareness principles into complex clinical scenarios 

11. Negotiate patient-centered care among multiple care providers. 

12. Appreciate roles of a variety of health care providers, including but not limited to consultants, therapists, 
nurses, home care workers, pharmacists, and social workers. 

13. Work effectively as a member within the interprofessional team to ensure safe patient care. 

14. Consider alternative solutions provided by other teammates. 

15. Demonstrate how to manage the team by using the skills and coordinating the activities of interprofessional 
team members. 

1 6. Recognize health system forces that increase the risk for error including barriers to optimal patient care. 

17. Identify, reflect on, and learn from critical incidents such as near misses and preventable medical errors. 

1 8. Dialogue with care team members to identify risk for and prevention of medical error. 

19. Understand mechanisms for analysis and correction of systems errors. 

20. Demonstrate ability to understand and engage in a system-level quality improvement intervention. 

21. Partner with other health care professionals to identify, propose improvement opportunities within the 
system. 
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reviewed and grouped into its six core, defining 
principles (i.e., system consultant, care coordinator, 
resource manager, patient advocate, team collabora- 
tor, and system evaluator). If any core principle is 
not represented, additional material needs to be 
created to fill the gap. 

2. The curricular objectives should be driven by 
clinical outcomes. 

After categorizing the content, examples of clinical 
behaviors or patient outcomes can be linked accord- 
ingly. The categorization should align with not 
just the six basic principles of SBP but also the 21 
milestones (3) that map to the core competency 
of SBP (Table 4). For example, being a competent 
system consultant means being able to navigate your 
patient through a complete Medicaid application 
or to assist an elderly patient choose a prescription 
drug plan under Medicare Part D. Assistance 
with this real clinical experience could map to the 
milestone of 'Reflect awareness of common socio- 
economic barriers that affect patient care' or, 
involving the social worker to complete this task, 
to the milestone of 'appreciate role of various health 
care providers . . . including social workers'. Under 
team collaborator, a resident may consult with a 
diabetic nutritionist to create a culturally tailored 
diet that helps improve a diabetic's glucose control. 
The examples would then be listed for each category 
of SBP and its corresponding milestone (Table 4). 

3. The teaching modalities should be interactive and 
clinically relevant. 

The concrete examples outlined above should be in- 
corporated into a curriculum via a teaching strategy 
that is interactive and clinically based. Some didac- 
tic sessions may be warranted, but other pedagogies 
should be considered. Of course, the degree of 
innovation will depend on institutional resources, 
but educational technology such as podcasts (if 
available) can often overcome scheduling or time 
barriers. Other examples include case-series, peer- 
assisted learning, simulations, and direct observa- 
tions. The teaching and learning modality should 
also inform the evaluation process. 

4. The assessment of competency should utilize mile- 
stones and/or EPAs. 

Attention to the course evaluation should be turned 
from content-oriented to assessment-based, and eva- 
luative tools should focus on patient outcomes or 
physician behaviors - such as chart reviews, patient 
satisfaction, or Plan, Do, Study, Act (PDSA) cycles. 
For example, in addressing the 'resource manager' 
component, one could conduct chart reviews to see 
how often residents are asking about patients' 
abilities to manage the costs of medications and, 
when a problem is identified, how the resident 



addressed the issue. For the 'team evaluator' aspect, 
a 360 degree evaluation of a resident involving 
ancillary staff, social workers, and nurses could be 
conducted to measure specific physician behaviors 
toward members of the interprofessional team. 
Other assessment tools that could be considered 
are direct observation of SBP skills - such as ob- 
serving, scoring, and documenting resident-patient 
encounters around the use of interpreters. There are 
many other examples; emphasis should remain on 
curricular objectives driving by clinical outcomes 
and tied to a milestone (Table 4) or EPA. 

Conclusion 

Many of the lessons learned in reformatting our own SBP 
curriculum centered on best delivery of teaching methods 
and assessment tools - both of which should align with 
current changes in health care and health professions 
education. Despite not yet being able to share successful 
experiences from our reformatted curriculum (7), sharing 
lessons learned may assist other programs in enhancing 
or creating their own SBP curriculum as mandated by 
the ACGME. We believe our experience, this literature 
review, and the guiding principles provided can assist in 
the development of competency-based curricula. 
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